Combinatorial and Random Testing Effectiveness for a Grid Computer Simulator

This paper compares the effectiveness of random and t-way combinatorial testing, where t = 2, 3, 4, for a grid computer network simulator. Previous investigations of random vs. combinatorial testing have reached conflicting results, with some showing more effective fault detection for combinatorial testing and others finding no significant difference between the two methods. In this paper, these two methods are compared for deadlock detection on a simulator with tests covering 2-way to 4-way combinations of configuration values, paired with an equal number of randomly generated tests. Random testing provided better results than pairwise (2­ way) testing and there was no statistically significant difference between the methods for 3-way testing, but 4-way combinatorial tests detected more deadlocks than the same number of random tests. The paper reviews explanations for these results and implications for testing.